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ABSTRACT 

The paper describes the delivery of the courses in the framework of the project implementation 
and presents the effect the change in the methodology had on student performance as measured 
by final grade. Methodology: University of Pristina piloted blended courses in 2013 under the 
framework of the Tempus BLATT project. The blended learning approach was tackled through a 
supplemental model where online component was added to unreduced seat time of the courses. 
A total of 791 students were observed. Student performance and attainment of learning objec¬ 
tives in traditional vs. blended courses were compared. Notwithstanding the improved students' 
performance in some of the courses, the overall grades were not significantly higher with the in¬ 
troduction of blended approach (p>0.05). Despite the divergence from the Gaussian curve, show¬ 
ing the prevailing low grades, teachers tend to estimate the attainment of the learning objectives 
favorable. The differences between final grades and learning objectives score ranks were highly 
significant. 
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INTRODUCTION 

A variety of terms are used in the literature to define ‘hybrid learning', with many of these being 
synonymous to one another such as: mix-mode, melted, multi-method, integrated, blended, flexi¬ 
ble, hybrid, e-learning, etc. However, the multitude of labels for a notion indicate that “no domi¬ 
nant model has yet been accepted as a definition of standard practice” (Dziuban, Hartman, & 
Moskal, 2011). Terms ‘blended’ and ‘hybrid’ seem to be the most frequent in literature and are 
often alternately used. Nonetheless, Friend (2013) points out a distinction between the two terms 
that complement our implementation of the approach within the Tempus BLATT project. 

In other disciplines, the term hybrid is far more common, with blended being used parenthetically 
just to make sure readers know what authors are talking about. Rather than being concerned with 
how to structure the specifics of a course, these fields are interested in helping students master 
the subject matter. To them, each delivery mode serves as a tool, allowing them to instruct 
students a different way. They emphasize the use of whatever delivery mode is most appropriate 
for the task that needs to be accomplished. (Friend, 2013) 

The hybrid approach is rendered on a wide scale springing from the synthesis of various 
instruction methods with or without using technology (Torrisi-Steele, 2011; Verkroost, Meijerink, 
Linsten, & Veen, 2008). In addition, Shank (2011) defines it as a way to support learning with 
additional tools and resources, or as Purnima (2002) describes it: “learning that mixes various 
event-based activities”. In contrast, other definitions signify the use of digital technology to 
improve student engagement and reduce seat time (Garrison & Vaughan, 2001). Dziuban et al. 
(2004) interpret the approach as the one that combines communal aspect of physical classroom 
with “active learning possibilities” of the online component. These authors are not interested in 
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“the ratio of the delivery modalities” but rather in the remodeling of the teaching and learning 
processes. 

Hybrid learning and its effectiveness compared to the traditional course delivery is the focus of 
this paper; we have piloted our hybrid courses under the supplemental model which “retains the 
basic structure of the traditional course and uses technology resources to supplement traditional 
lectures and textbooks” (Westminster College, 2011). This model is based on the principle that 
synchronous oral communication is integrated with asynchronous written communication online to 
create “a unique learning experience congruent with the context and intended educational 
purpose” (Garrison&Vaughan, 2001). 


METHODOLOGY 

The comparison of the mean final grades and the achievement of the learning outcomes were 
investigated in the comparable courses delivered by the hybrid and the traditional face-to-face 
methodology. The courses were delivered in two consecutive school years (2012/13 and 
2013/14, respectively), encompassing two generations of students, where the hybrid courses fol¬ 
lowed the traditional ones. 

The implementation of the hybrid approach at the University of Pristina, Serbia was performed 
under the Tempus BLATT (Blended Learning: Advanced Teacher Training) project 1 . Its primary 
focus is the systematization of teaching with digital technology at the University. Moreover, the 
project hinges on the introduction of student-centred learning, as well as the revision of the 
European Credit Transfer System (ECTS) in the given context (Tempus BLATT, 2013). The 
project aims, however, not only to engraft new methodology, but to induce thorough changes of 
the curricula entangled, which are expected to become the model for the wider curricular reforms 
throughout the university. 

The courses subjected to the analysis in this paper were added an online component not 
detracting from the contact hours prescribed by the study programme. The online component, 
virtual classroom, was realized through Moodle. The virtual classroom contains a comprehensive 
syllabus, course materials, and, depending on the nature of the course itself, activities and 
assignments that would supplement class work. The evaluation of the students' achievement 
encompassed the work they did in the virtual classroom. It was an integral part of the course. 

Lecturers were supported in transforming their courses into hybrid by e-learning specialists from 
partner institutions. They implemented hybrid methodology with the significant part (one fourth to 
one third of the total) of the learning materials delivered online, while preserving the existing 
structure of the face-to-face lessons. Virtual training sessions in using Moodle, creating course 
materials, using open educational resources, and designing a course were organised via Adobe 
Connect software. In total, seven courses were piloted in the fall semester 2013/14: Biology of 
exercise at the Faculty of Sports and Physical Education; Project Management at the Faculty of 
Economy; Contemporary English Language 1 at the Faculty of Philosophy; IT in Environmental 
Protection at the Faculty of Technical Science; and three courses at the Faculty of Medicine, 


1 Tempus BLATT 530266-TEMPUS-1-2012-1-XK-TEMPUS-JPCR 
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Endocrinology, Practical exercises in Paediatrics , and ECG Basics. However, the optional IT in 
Environmental Protection course was canceled due to the low interest of the students. The 
courses were externally evaluated by the partner institutions (Tempus BLATT, 2014). Six of the 
seven piloted courses are the focus of the statistical analysis in this paper because the course IT 
in Environmental protection was cancelled due to low enrolment. 

A total of 791 students were encompassed by the present study, 417 of whom took the traditional 
courses and 374 involved in the hybrid ones. Both generations of students had the same faculty 
admission criteria, implying the equivalent average success in high school and the equivalent 
entrance exam scores; they were highly matched concerning the conditions related to and the 
success they showed in the studies prior to the start of the respective courses. The age and the 
gender distribution between the two generations were not different. Moreover the course 
instructors were the same, and the assessment criteria for the two generations of the students 
were equal. 

The statistical evaluations of the mean final grades and the attainment of the learning objectives 
(as evaluated by the course instructors) were performed using non-parametric Mann-Whitney U 
test. The comparisons were performed for the individual courses, as well as the overall scores 
obtained from all courses delivered. 

The normality test was used for the estimation whether the data (e.g. final grades and learning 
objective scores) fit the Gaussian distribution, as well as for the determination of the divergence 
rate. 


RESULTS 

The comparison of the mean final grades in hybrid and traditional courses, comprising all 
participating students, showed a slightly better student achievement in the hybrid courses. 
However, the differences were insignificant as shown by the results of the statistical test analysis 
(Table 1). 


Table 1: The results of the Mann-Whitney U test of the difference in final grades between the 
hybrid and traditional courses. 


Method of teaching 

The mean final grades 

Traditional 

6.88 +- 1.44 

Hybrid 

7.03 +- 1.55 

Test Statistics 

Mann-Whitney U 

7.22 X 10 4 

Wilcoxon W 

2.02 X 10 b 

Z 

-0.074 

p (2-tailed) 

0.931 

p >0.05 



The results of the dissection of the individual courses are shown in the Table 2. The students in 
the three of the six courses had considerably (p<0.05) improved their achievements with the 
introduction of hybrid pedagogy, while no significant differences were registered in the three other 
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courses. Notably, the utmost improvement was achieved in those courses where preparation and 
planning of the lessons was neglected in the past; two of them were medical courses which 
traditionally rely primarily on personal level of scientific and medical knowledge and the skills of 
the instructors. Therefore, the introduction of the hybrid pedagogy was the opportunity for these 
instructors to, for the first time, develop a detailed plan of the course, including specific learning 
objectives, diverse methodologies, different assessment forms, etc. This was achieved through 
closely monitored alignment process and the implementation of wider theories of learning and 
through changes in assessment methodology. Previously, the lecturers were sovereign in 
curriculum design and course delivery and were not subject to evaluation and quality assurance 
procedures. 

The similar may be inferred for the Project Management course. Therefore, it appears that the 
use of diversified pedagogical methods initiated by the creation of a hybrid course contributed to 
the improvement of the delivery, rather than the use of digital technology alone. However, this is 
to be confirmed in a subsequent multiple regression analysis, which will be the subject of a 
different study. 


Table 2: The results of the Mann-Whitney U test of the differences in the final grades between 
the traditional and hybrid courses. 


The course name 

Traditional 2012/13 

Hybrid 2013/14 

Mann - Whitney 


mean final grade 

mean final grade 

U test results 

Biology of exercise 

6.62 +- 0.91 

6.93 +- 1.29 

p=0.08 

Project Management 

6.35 +- 1.32 

6.89 +- 1.65 

p=0.03 

Contemporary English Language 1 

6.86 +- 1.18 

6.87 +- 1.79 

p=0.96 

Practical Exercises in Paediatrics 

6.77+-1.59 

6.35+-1.7 

p=0.27 

The ECG Basics 

5.64+-1.06 

6.33+-1.45 

p=0.03 

Endocrinology 

7.077 +- 1.57 

8.12 +- 1.12 

p=0.0001 


Considering the high number of failures and low scores noticed both face-to-face and hybrid 
courses, we tested the normality of the distribution of the final grades. The distribution of the 
grades on both the traditional and hybrid courses demonstrated an evident deviation from the 
normal Gaussian curve (Figures 1 and 2, respectively). 
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Histogram 



Figure 1: Distribution of final grades in traditionally delivered courses. 


Histogram 



Figure 2: Distribution of final grades in hybrid courses. 


Low scores (5, 6 and 7) far exceeded good and excellent ones with both methods of teaching. 
The divergence between the normal and the observed distributions of the final grades was highly 
significant (p<0.01) with both types of courses, as evidenced by the results of the statistical 
normality tests (Tables 3 and 4, respectively). 


Table 3: Test of normality of the distribution of final grades in traditional courses 



Test of normality: Traditional courses 


Kolmogorov-Smirnova 


Shapiro-Wilk 

Statistic 

df p Statistic 

df p 

0.152 

508 1,09x 1 O' 31 0.913 

508 1,67 XI 0' 1B 
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Table 4: Test of normality of the distribution of final grades in hybrid courses 


Kolmogorov-Smirnova 

Test of normality: Hybrid courses 

Shapiro-Wilk 

Statistic 

df p Statistic 

df 

P 

0.239 

222 1.07 x10” 45 0.893 

281 

2.87 x 10" 13 


The mean evaluation scores of the attainment of learning objectives, as assessed by the 
instructors, were significantly better with the hybrid pedagogy comparing to pure face-to-face 
delivery. Since the subsequent analysis revealed the non-Gaussian distribution of the scores, the 
use of the non-parametric Mann-Whitney U test was mandatory. Both T- and Mann-Whitney U 
tests showed highly significant differences in achieving the predicted learning objectives (Table 5, 
respectively). 


Table 5: The results of the Mann-Whitney U Test of the differences in the attainment of learning 
objectives between the traditional and hybrid courses (as assessed by the course instructors). 


Method of teaching 

The mean learning objectives scores 

Traditional 

3.42 +- 0.61 

Hybrid 

4.06 +- 0.61 

Test Statistics 

Mann-Whitney U 

1404 

Wilcoxon W 

22.5 

Z 

-5.21 

p (2-tailed) 

0.0001 

p < 0.001 



The results of the instructors’ evaluation of the attainment of learning objectives in their own 
individual courses (1-5 scale, Table 6) also show the unanimous impression that these objectives 
were more advantageously achieved, comparing to the previous traditional courses; these 
convictions were confirmed statistically, by the results of the Mann-Whitney U testing of the 
differences between the evaluation results in hybrid vs. traditional courses. 

Despite the fact that the teachers highly valued the improvement in attainment of the course 
objectives with the introduction of the new methodology, the relatively low mean grades in the 
majority of the courses contradicted the perceptions that the learning objectives were almost 
completely accomplished. The dissociation between the teachers' expectations and evaluation of 
their own efforts in creating and delivering the course contents and the students' achievements 
on the final exam was already marked with the traditional courses and even more so with the 
introduction of the hybrid methodology (see Figure 5). Moreover, the students fail to show 
significant improvement on the final exam in three of the six courses. 
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Table 6: The results of the Mann-Whitney U tests of the differences in the attainment of the 
learning objectives between the individual traditional and hybrid courses (as assessed by the 
course instructors). 


The course name 

Traditional course 
2012/13 mean 
learning objectives 
grade 

Hybrid course 
2013/14 mean 
learning objectives 
grade 

Mann - Whitney 

U test results 

Biology of Exercise 

3.72 +- 0.56 

3.97 +- 0.50 

p=0.03 

Project Management 

2.66 +- 0.81 

4.33 +- 0.52 

p=0.03 

Contemporary English Language 1 

3.14 +- 1.21 

3.96 +- 0.89 

p=0.01 

Practical Exercises in Paediatrics 

3.33 +-0.81 

4.33 +-0.51 

p=0.03 

The ECG Basics 

3.08+-0.66 

3.83+-0.41 

p=0.0001 

Endocrinology 

3.08 +- 0.66 

4.5 +- 0.54 

p=0.006 


Moreover, the test on the normality of the distribution of the instructors’ learning objectives 
evaluation scores (Figures 3 and 4, Tables 7 and 8, respectively) showed a highly significant 
divergence from the Gaussian curve. Contrary to the final grades, however, the distribution of the 
learning objective scores deflected towards the highest levels with both methods of delivery, but 
especially with the hybrid. 


Histogram 



Mean *3 42 

Std Oov -0 777 
N -67 


Figure 3: The distribution of the learning objectives scores in traditional courses 
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Histogram 



Figure 4: The distribution of the learning objectives scores in hybrid courses 


Table 7: Test of normality of the learning objectives attainment scores in the traditional courses 


Test of Normality: Traditional courses 
Kolmogorov-Smirnova Shapiro-Wilk 


Statistic 

df 

P 

Statistic 

Df 

P 

0.325 

67 

1.08 x 10' 19 

0.776 

67 

1 x 10' 08 


Table8: Test of normality of the learning objectives attainment scores in the hybrid courses 


Test of Normality: Hybrid courses 
Kolmogorov-Smirnova Shapiro-Wilk 


Statistic 

df 

P 

Statistic 

df 

P 

0.354 

75 

2,04 x lO'* 6 

0.727 

75 

1 x 10 1U 


Upon these findings, we ranked the learning objective scores, translating them in the 5-10 scale 
system in order to compare them statistically to the student final grades in the respective courses 
(table 9). The difference between these assessments was highly significant (P < 0.001), as 
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expected with the previous results. On the other hand, the traditional courses showed no 
significant diversity between the attainment of the objectives and the final grades. 


Table 9: The system of ranking of the evaluation of the learning objectives with the aim of 
translation from 1-5 to 5-10 scale and comparing them with the final grades in the respective 
courses. 


Score 

Rank 

5 

10 

4.5 

9 

4 

8 

3 

7 

2 

6 

1 

5 


There is a significant dissociation between the instructors’ evaluation of the learning objectives 
that may be achieved in their hybrid courses, as well as knowledge and skills that students 
demonstrate in the exam (Table 10, Figure 5). This difference was a result of the exaggerated 
expectations of the instructors with the use of the new methodological approach. By the results of 
our study the changes in methodology of teaching alone, apart from the simultaneous 
modification in assessment methodology and the adjustment of the learning objectives, were 
apparently not sufficient to support tendencies towards more efficient learning. 


Table 10: The results of the Mann Whitney U test of the differences between the student final 
grades and the learning objectives scores (as assessed by the course instructors) in the hybrid 
courses. 


Test Statistics 

Grades vs. Objectives 

Mann-Whitney U 

5511.5 

Wilcoxon W 

46266.5 

Z 

-6.618 

p (2-tailed) 

0.0001 

P < 0.001 
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Figure 5: A comparative view of the student final grades and the learning objectives scores (as 
assessed by the course instructors) 


In accordance with the concept of constructive alignment, if the learning outcomes within a given 
course or study program were not accomplished in a satisfactory way, the instructor should put 
additional effort in changing either learning methodology or assessment methodology or (in case 
neither of these succeed) the learning outcomes themself. The instructor must take his share of 
responsibility for accomplishing the learning objectives, for the overall success in the exam, 
diversifying and modernizing the assessment methodology and, in particular, the introduction of 
regular evaluation of the objectives, learning methodology and assessment methods, instead of 
holding students responsible for any decline and inefficiacy in the process; this could be a course 
towards the optimization of the studies and acceleration of the higher education reforms. Knowing 
that repeated changes in Serbian legislation concerning Higher Education in recent years had 
little effect, it is obvious that this kind of bottom-up incitement of the reform process may be more 
than welcome. 


DISCUSSION 

The review of the current research studies indicates that the hybrid learning “is now the preferred 
model for online course design” (Precel, Eshet-Alkalai, & Alberton, 2009). These studies suggest 
apparent dominance of hybrid over online learning for multiple reasons such as loneliness and 
social isolation, lack of cognitive skills, and lecture format being all too familiar. These issues are 
resolved in hybrid learning by combining brick and mortar classroom with its opportunitites for 
social contact and online component of the course with its opportunities for collaborative learning 
(Means, Toyama, Murphy, & Baki, 2013). According to Kitchenham (2011), hybrid learning can 
achieve the goals and objectives of any higher education institution in the world. In addition, 
Dziuban et al. (2004) believe that hybrid courses can be the solution to the problems of “cost, 
access, efficiency, and timely degree completion”. 

Banarjee (2011) poses a potent question whether the trend of increasingly moving to hybrid and 
fully online delivery of the courses in the tertiary education is “a desirable trend for smaller 
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institutions”. She believes that learning effectiveness and student satisfaction are crucial in 
overcoming problems in “transforming face-to-face teaching to the technology mediated learning 
environments”. 

In addition, hybrid learning is not considered a transitional step toward online learning. On the 
contrary, the approach is “a discrete option which institutions choose on its own merits” (Allen & 
Seaman, 2006). The authors also note: 

Most Chief Academic Officers believe that the quality of online instruction is equal to or superior 
to that of face-to-face learning. In 2003, 57% of academic leaders rated the learning objectives in 
online education as the same or superior to those in face-to-face. That number is now 62 percent, 
a small but noteworthy increase. Also, the proportion of those who believe that the online learning 
objectives are superior to those for face-to-face is still relatively small but has grown by 4 percent 
since 2003 from 12.1 percent to 16.9 percent, (p. 15) 

In accordance to our results, the study by Willson et al show that “the withdrawal rates and failure 
rates were not significantly different between the two modes of course delivery” (Willson & Allen, 
2011). However, other literature data (Capra, 2011) indicate that students undertaking study 
online or at a distance tend to have higher withdrawal rates. Moreover, the Montana State Uni¬ 
versity study, which compared two versions of the same required electrical engineering course, 
using exactly the same instructional materials, indicates that student learning was not compro¬ 
mised in the online delivery version of the course. Plumb and LaMeres (2011) report: 

Two groups of students, whose overall college grade point averages were no different 

entering the two versions of the course, received nearly an identical average final grade 

in the course and performed similarly on individual graded components in the course, (p. 

7) 

The United States Department of Education (2009) reports that there is some evidence that 
distinguishes blended learning in comparison to traditional or online teaching. The Department 
surveyed several hundred empirical researches and singled out 51 studies which measured 
learning objectives in the three learning modes. According to these studies, online and blended 
learning environments yielded better results, with blended learning performing significantly better. 
These results “are attributable in part to active learning strategies, which include opportunities for 
reflection and interaction with peers, and in part to the enriched content that characterizes well- 
designed online and blended courses” (U.S. Department of Education, 2009). McGee and Reis 
(2012) inform on common focus of such research studies “learner traits, grades, faculty 
member/learner levels of satisfaction, and/or levels of learner engagement”. 

A division of the Education Department that rates classroom curriculums states that “educational 
software is not an improvement over textbooks” (U.S. Department of Education, 2009). It has 
been noted that hybrid approach encompasses additional learning time, instructional resources, 
and course elements that encourage interactions among learners which leads to a conclusion 
that the combination of these elements is instrumental in advantages of the approach. Our results 
also pointed out that the success of the delivery was related mostly to the extent of the lesson 
preparation, clarity of the learning objectives, the selection of the additional materials prepared, 
the amount of time spent in lesson preparation, the diversity of the methodologies used; the 
courses with least pedagogical preparation in the traditional mode improved mostly when those 
elements were required of them during the creation of the hybrid course. Finally, Mean, Toyama, 
Murphy, and Baki (2013) emphasize the need for experimental testing of the course design for 
different types of learners. The U.S. Department of Education report (2009) further attests to this 
view: 
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Despite what appears to be strong support for blended learning applications, the studies in this 
meta-analysis do not demonstrate that online learning is superior as a medium. In many of the 
studies showing an advantage for blended learning, the online and classroom conditions differed 
in terms of time spent, curriculum and pedagogy. It was the combination of elements in the 
treatment conditions (which was likely to have included additional learning time and materials as 
well as additional opportunities for collaboration) that produced the observed learning 
advantages. At the same time, one should note that online learning is much more conducive to 
the expansion of learning time than is face-to-face instruction, (p. 18) 

In the 2011 New York Times Article “In Classroom of Future, Stagnant Scores” by Matt Richtel, 
the 2009 report was opposed: 

... they [digital technologies] may simply amplify what's already occurring — for better or worse. 
Good teachers can make good use of computers, while bad teachers won't, and they and their 
students could wind up becoming distracted by the technology. 

Furthermore, Weber and Lenon (2007) came to a conclusion that students perform equally in 
both formats. Their analysis on means GPA (grade point average) showed that “course format 
had an insignificant contribution to learning achievements”. Previous knowledge and academic 
abilities were instrumental in deterring student performance in both course formats. 

In a study “Blended Learning vs. Traditional Classroom Settings: Assessing Effectiveness and 
Student Perceptions in an MBA Accounting Course (Chen & Jones, 2007), in the multiple 
regression analysis, three of 12 actors were selected as the most dominant: “the quality of the 
instructor, the clarity of the instruction and the density of the course matter”. Additionally, meta 
skills needed for better attainment of learning objectives were analytical and computer skills. 

In consistency with our findings, Nagel (2009) infers that appropriate mixture of instructional 
methods brings good results. He also adds that “jumping on the latest bandwagon or applying a 
single strategy across your whole course will seldom be successful”. Trends in e-Learning report 
(2006) stipulates that common issues arising from the growing number of instances of online 
delivery are “support staff needed for training and technical assistance, orientation/preparation for 
taking distance education classes by students and workload issues”. 

The report of the study designed in a similar fashion with ours, is based upon a pilot-test of the 
implementation of four blended courses taught at the Harvard College in the fall of 2013 (Rutter, 
2104). Their interpretation of blended learning is “the integration of online or other components 
into a residential educational experience” (Rutter, 2014). All the courses were previously taught 
face-to-face, although they varied in subject matter, course design, and instructional methods. 
Ruttler (2014) reports: 

As many students were new to the concept of blended learning, they responded more to the 
course structure and execution than to the efficacy of specific online or blended elements. 
Students appreciated the high quality of the materials and a majority found them to be interesting 
and engaging. For the most part, students spent relatively the same amount of time outside of 
class (homework, preparation) in the blended versions as they would in traditional Harvard 
courses. Students valued the increased flexibility and ability to learn at their own pace, but still 
wanted meaningful in-person interactions with faculty and among themselves, (par. 11) 
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CONCLUSION 

In conclusion, our study showed that the total student achievement in all six observed courses did 
not statistically differ among the blended and face-to-face instruction. Three of the blended 
courses - two medical courses and one at the Faculty of Economy - showed a significantly 
(p<0.05) better final grades compared to the traditional delivery. The courses where the 
improvement was most prominent were the ones where the lesson preparation in the traditional 
approach was the least adequate. The invested effort in preparation, clearer objectives and 
stronger connection between the learning objectives and the assessment expectations were 
instrumental in better student accomplishment, not the application of hybrid approach itself. 

In all hybrid courses, teachers tended to overestimate the attainment of learning objectives (which 
were, according to their subjective evaluation, significantly better achieved with the use of hybrid 
methodology) and this was in sharp discrepancy with the student final grades. The constructive 
alignment of learning objectives, learning activities and assessment methods may be the best 
strategy to overcome this disparity and improve course delivery, the hybrid inasmuch as the 
traditional. 
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